fix: reconcile pending bundles on a timer when blocks stall (SUP-1552) - #50

Part of SUP-1552. Closes ZER-928

Problem

Chain 55516 (Geo testnet) stalled for 19h45m on 2026-08-13/14. eth_sendUserOperation kept returning 200 OK; no userOp ever got a receipt. The same pattern has recurred since June: 18d, 7.5d, 7d, 5.6d, 4.3d, plus a dozen 20–50h gaps.

It is a deadlock, and the bundler is both halves of it.

handleBlock is the only path that resolves a submitted bundle: it reads receipts, applies resubmitStuckTimeout, rotates stuck bundles, and returns the executor wallet to the sender pool. Its only trigger is watchBlocks({ onBlock }), which fires on a new block.

Arbitrum Orbit chains produce a block only when they receive a transaction. On a low-traffic testnet the bundler is effectively the only writer, so:

sends start failing (upstream rate limit)
        ↓
bundles left tracked, wallets held
        ↓
no wallet free  ───────────────┐
        ↓                      │
bundler can't submit           │
        ↓                      │
no txs on chain                │
        ↓                      │
no new blocks                  │
        ↓                      │
onBlock never fires            │
        ↓                      │
handleBlock never runs         │
        ↓                      │
wallets never returned ────────┘

getWallet() is an unbounded spin (createRedisSenderManager.ts:80), so every later bundle parks there forever. resubmitStuckTimeout is a time condition evaluated only on a block event, so the escape hatch is unreachable exactly when it is needed.

Nothing inside the loop can break it. What actually broke it was the customer sending an unrelated transaction from outside; the relayer resumed one second after that block landed.

Evidence (BetterStack, Ultra Relay (Prod), chainId 55516)

mempool-store logged userOps going outstanding -> processing throughout the stall, then nothing: they reached sendBundleToExecutor and parked in getWallet(). Zero warns or errors for 20h, which is the tell that handleBlock never executed once. The trigger was upstream rate limiting on eth_sendRawTransaction at 23:16–23:29 ({"code":-32017,"message":"Rate Limit Exceeded..."}).

Change

1. Stale-block watchdog. watchBlocks stays: receipt polling is correctly block-driven, since a receipt cannot change without a new block and each tick costs one receipt lookup per pending bundle plus gas price reads. Replacing it with an interval would have raised RPC load on the very endpoint that was rate-limiting us.

Instead, a timer re-arms only the time-based check that the block gate makes unreachable. It calls handleBlock when no reconcile has happened within resubmitStuckTimeout and bundles are still pending. On a healthy chain a block arrives every blockTime, so lastReconcileAt stays fresh and the watchdog does zero RPC. It fires a staleBlockWatchdogFired warn when it does act.

This is the same fix reconcileQuarantinedWallets already applies one layer down, for the same reason: that code path's comment notes that waiting on latest > stuckNonce would deadlock, because the benched wallet sends nothing. Same bug, one level up. No new config option; resubmitStuckTimeout and blockTime already exist.

2. currentlyHandlingBlock held in try/finally. It was set at the top of handleBlockInner and cleared only on the success path. A throw in there (a store error in freeSubmittedBundle, say) left it set, so every later tick returned early at the overlap guard and reconciliation never resumed for the life of the process. That would also have silently defeated the watchdog above, so it is fixed here rather than separately.

Tests

src/executor/executorManager.test.ts, 4 cases with fake timers. Each was confirmed to fail with the corresponding fix reverted:

Test Files  5 passed (5)
     Tests  27 passed (27)

pnpm lint: 0 errors (the 902 warnings are pre-existing on main).

Not in this PR

• Upstream rate limiting. Chain 55516 points at the public Conduit endpoint with no API key, and -32017 was still firing on Aug 25 (02:36, 02:44, 03:21, 06:45, 08:01). This PR stops it from causing a multi-day outage; it does not stop the sends from failing. Needs an API key on the deployment.
• getWallet() timeout. The unbounded spin at createRedisSenderManager.ts:80 is a landmine regardless. Worth bounding as defense-in-depth, but it treats starvation rather than the cause, so it is left out here.